Papers with *Stochastic Tuning*
Stochastic Fine-Tuning of Language Models Using Masked Gradients (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are the dominant paradigm in Natural Language Processing but fine-tuning them for specific downstream tasks often requires updating a vast number of parameters. |
| Approach: | They propose a method that selectively updates a small subset of parameters in each step of the tuning process. |
| Outcome: | The proposed approach outperforms existing fine-tuning methods while updating merely **0.08**% of the model’s parameters. |